Troubleshooting network and TCP/UDP port connectivity issues on ESX/ESXi(2020669)
Posted on by Admin
Related Post
- Helping Public Sector Organisations Define Cloud Strategy
- How to change the VLAN ID of the Service Console in ESX from the command line/console
- Cisco UCS and Vmware Interfaces (Vnics) HA Design Considerations
- vSphere Client Parameters
- Logging in to the vSphere Client or performing a search in the vSphere Web Client fails with the error: Client is not authenticated to VMware Inventory Service (2080396)
Troubleshooting network and TCP/UDP port connectivity issues on ESX/ESXi(2020669)
Purpose
This article provides information on troubleshooting network and TCP/UDP port connectivity issues using these troubleshooting tools:
- ping/vmkping to troubleshoot network connectivity between two servers.
- telnet to troubleshoot TCP port connectivity.
- netcat (nc) to troubleshoot TCP port connectivity.
- openssl to troubleshoot SSL port connectivity and verify SSL certificate information.
- tcpdump and tcpdump-uw to collect packet traces to troubleshoot network issues.
- netstat and esxcli network to view active TCP/UDP connections to the host.
Note: The telnet and nc tools help you to check if a TCP port is online or if there is a firewall blocking access to a TCP port.
Resolution
Confirming network connectivity with ping and vmkping
To check if a remote host is online, you can use the ping and vmkping commands on ESX/ESXi host. The syntax of these commands are:# ping destination-ip
# vmkping destination-ip
You see an output similar to:
# vmkping 192.168.48.133
PING 192.168.48.133 (192.168.48.133): 56 data bytes
64 bytes from 192.168.48.133: icmp_seq=0 ttl=64 time=0.978 ms
64 bytes from 192.168.48.133: icmp_seq=1 ttl=64 time=1.009 ms
In this sample output, you can see that the ESX/ESXi host is able to communicate with the remote host with IP address 192.168.48.133.
Note: On ESX hosts, the ping command is run from the network stack of the Service Console, while the vmkping command is run from the vmkernel network stack, which is independent of the Service Console. On ESXi, the ping and vmkping are the same command and run from the vmkernel network stack because there is no Service Console in ESXi.
For more information on using the ping command, see Testing network connectivity with the ping command (1003486).
Confirming connectivity to a TCP port with telnet
Note: Telnet is available only on ESX hosts. For ESXi 3.5, 4.x and 5.x, you will need to use the netcat (nc). Please see the section below titled “Confirming connectivity to a TCP port with netcat” for further information.While the ping command confirms connectivity, it does not necessarily mean that all TCP ports on the remote host can be reached. It is possible for a network firewall to allow or block access to certain ports on a host.
To check if specific TCP ports are running on the remote host, you can use the telnet command to confirm if a port is online.
# telnet destination-ip destination-port
When trying to establish a telnet connection to TCP port 80, you see an output similar to:
# telnet 192.168.48.133 80
Trying 192.168.48.133…
Connected to 192.168.48.133.
Escape character is ‘^]’.
In this sample output, you can see that you are connected to port 80 (http) on the server with IP address 192.168.48.133.
If you choose a port number for a service that is not running on the host, you see an output similar to:
# telnet 192.168.48.133 81
Trying 192.168.48.133…
telnet: Unable to connect to remote host: Connection timed out
In this case, you can see that there is no response when you attempt to connect to port 81 on the server 192.168.48.133.
Note: Telnet is an application that operates using the TCP protocol. UDP connectivity can not be tested using Telnet.
Confirming connectivity to a TCP port with netcat
The telnet command is not available in any versions of ESXi and, therefore, you must use netcat (nc) to confirm connectivity to a TCP port on a remote host. The syntax of the nc command is:# nc -z <destination-ip> <destination-port>
When testing connectivity to TCP port 80, you will see an output similar to:
# nc -z 192.168.48.133 80
Connection to 192.168.48.133 80 port [tcp/http] succeeded!
In the sample output, you can see that you are able to establish a connection to TCP port 80 on the host 192.168.48.133.
Note: Netcat includes an option to test UDP connectivity with the -uz flag, but because UDP is a connectionless protocol, it will always report as ‘succeeded’ even when ports are closed or blocked. Instead, test bi-directional UDP connectivity using tcpdump or tcpdump-uw.
The nc command can also be used to check the connectivity to a range of TCP ports on a remote host:
# nc -w 1 -z 192.168.48.133 20-81
Connection to 192.168.48.133 22 port [tcp/ssh] succeeded!
Connection to 192.168.48.133 80 port [tcp/http] succeeded!
The -w option specifies a timeout value.
Note: Port scanning is a very powerful troubleshooting tool, but may be against your company network or security policies. Check with your network or security team to ensure that they are aware of this activity.
Testing SSL port connectivity and certificate information with openssl
To test SSL ports, you can use the openssl command to test connectivity and also to confirm the current SSL information. This can be useful when confirming SSL certificates with vCenter Server. The syntax of the openssl command is:
# openssl s_client -connect destination-ip:ssl-port
You see an output similar to:
# openssl s_client -connect 192.168.48.133:443
CONNECTED(00000003)
Where 443 is the default SSL port.
In this sample output, you can see that connection to the remote server 192.168.48.133 over the SSL port was successful.
Note: The output may contain considerable information regarding the SSL certificates, which may be useful in troubleshooting certificate issues.
Collecting packet traces using tcpdump and tcpdump-uw
ESX and ESXi hosts come with the packet tracing tools, tcpdump and tcpdump-uw, which can be used to collect network traces. The network traces are useful in troubleshooting network issues.
- tcpdump can be used on ESX hosts to capture packet traces from the Service Console (vswif) interface.For more information on using tcpdump, see Verifying gateway IP connection using the tcpdump command (1008017).
To configure a vSwitch to monitor a virtual machine virtual network card, see Capturing virtual switch traffic with tcpdump and other utilities (1000880).
- tcpdump-uw can be used on ESXi hosts to capture packet traces from a vmkernel (vmk) interface.For more information on using tcpdump-uw, see Capturing a network trace in ESXi using Tech Support Mode or ESXi Shell (1031186).
Viewing active TCP/UDP connections with netstat and esxcli network
When troubleshooting network connectivity issues, it may be helpful to see all the active incoming and outgoing TCP/UDP connections on an ESX/ESXi host. ESX hosts can use the netstat command and ESXi 4.1 and later hosts can use esxcli network to show the list of TCP/UDP connections. The commands are:
ESX 3.5/4.x – # netstat -tnp
ESXi 4.1 – # esxcli network connection list
ESXi 5.0 – # esxcli network ip connection list
ESXi 5.1 – # esxcli network ip connection list
ESXi 5.5 – # esxcli network ip connection list
Sample output from an ESXi 4.1 host:
# esxcli network connection list
Proto Recv-Q Send-Q Local Address Foreign Address State World ID
tcp 0 52 192.168.48.136:22 192.168.48.1:55169 ESTABLISHED 0
tcp 0 0 127.0.0.1:62024 127.0.0.1:5988 TIME_WAIT 0
tcp 0 0 127.0.0.1:57867 127.0.0.1:5988 TIME_WAIT 0
tcp 0 0 127.0.0.1:62196 127.0.0.1:5988 TIME_WAIT 0
tcp 0 0 127.0.0.1:8307 127.0.0.1:52943 ESTABLISHED 5790
tcp 0 0 127.0.0.1:52943 127.0.0.1:8307 ESTABLISHED 5790
tcp 0 0 127.0.0.1:80 127.0.0.1:55629 ESTABLISHED 5785
tcp 0 0 127.0.0.1:55629 127.0.0.1:80 ESTABLISHED 6613
tcp 0 0 127.0.0.1:8307 127.0.0.1:56319 ESTABLISHED 5785
tcp 0 0 127.0.0.1:56319 127.0.0.1:8307 ESTABLISHED 5785
tcp 0 0 127.0.0.1:80 127.0.0.1:62782 ESTABLISHED 5166
tcp 0 0 127.0.0.1:62782 127.0.0.1:80 ESTABLISHED 6613
tcp 0 0 127.0.0.1:5988 127.0.0.1:53808 FIN_WAIT_2 0
tcp 0 0 127.0.0.1:53808 127.0.0.1:5988 CLOSE_WAIT 5166
tcp 0 0 127.0.0.1:8307 127.0.0.1:56963 CLOSE_WAIT 5788
tcp 0 0 127.0.0.1:56963 127.0.0.1:8307 FIN_WAIT_2 5785
tcp 0 0 127.0.0.1:8307 0.0.0.0:0 LISTEN 5031
tcp 0 0 127.0.0.1:8309 0.0.0.0:0 LISTEN 5031
tcp 0 0 127.0.0.1:5988 0.0.0.0:0 LISTEN 0
tcp 0 0 0.0.0.0:5989 0.0.0.0:0 LISTEN 0
tcp 0 0 0.0.0.0:80 0.0.0.0:0 LISTEN 5031
tcp 0 0 0.0.0.0:443 0.0.0.0:0 LISTEN 5031
tcp 0 0 127.0.0.1:12001 0.0.0.0:0 LISTEN 5031
tcp 0 0 127.0.0.1:8889 0.0.0.0:0 LISTEN 5331
tcp 0 0 192.168.48.136:427 0.0.0.0:0 LISTEN 0
tcp 0 0 127.0.0.1:427 0.0.0.0:0 LISTEN 0
tcp 0 0 0.0.0.0:22 0.0.0.0:0 LISTEN 0
tcp 0 0 0.0.0.0:902 0.0.0.0:0 LISTEN 0
tcp 0 0 0.0.0.0:8000 0.0.0.0:0 LISTEN 4801
tcp 0 0 0.0.0.0:8100 0.0.0.0:0 LISTEN 4795
udp 0 0 192.168.48.136:427 0.0.0.0:0 0
udp 0 0 0.0.0.0:427 0.0.0.0:0 0
udp 0 0 192.168.48.136:68 0.0.0.0:0 4693
udp 0 0 0.0.0.0:8200 0.0.0.0:0 4795
udp 0 0 0.0.0.0:8301 0.0.0.0:0 4686
udp 0 0 0.0.0.0:8302 0.0.0.0:0 4686
To retrieve errors and statistics for a network adapter, run this command:
# esxcli network nic stats get -n <vmnicX>
Where <vmnicX> is the name of a NIC in your ESXi host.
Additional Information
For more information about gathering information and troubleshooting network issues with esxcli network commands, see the Network Troubleshooting section in the vSphere Command-Line Interface Concepts and Examples guide.
